In safety-critical applications such as clinical decision making, it is important to implement safeguards preventing the use of incorrect predictions from computational models (Band et al., 2021; Challen et al., 2019). These safeguards rely on failure detection methods, which aim to automatically flag suspicious model predictions. For clinical deployment, reliable failure detection is critical for patient safety, enabling automatic referral to human experts (Kompa et al., 2021). As depicted in Figure 1, failure detection frameworks are typically divided in two stages: (i) confidence scoring (to quantify the likelihood of the prediction to be correct); (ii) a thresholding-step (to reject/refer samples with a low confidence score) (Corbière et al., 2019; Jiang et al., 2018; Band et al., 2021) We propose a new benchmark for evaluating in-domain failure detection in medical imaging classification models. Our experiments show that improved reliability against out-of-distribution inputs or model calibration does not necessarily translate to improved in-domain failure detection.